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Ethernet rules closed-loop system 



By John Edson and Wesley Cole 



The enterprise 
tool picks up 
speed and moves 
** % plant floor. 



Traditional process control systems have used 
programmable logic controller (PLQ-based cen- 
tralized control techniques to implement closed- 
loop control applications. Packet-based networks 
with collisions, such as Ethernet, are generally 
considered too slow and unreliable to safely han- 
dle closed-loop control. 

Today, various fieldbus technologies and the 
ability to inexpensively place significant compu- 
tation capability at the transducers allow control 
applications to be implemented using distributed 
techniques. This distributed function provides 
increased capacity, relieves computational and 
communication bottlenecks, and generally pro- 
vides more flexibility in system design, modifica- 
tion, and expansion. 

In a traditional process control system, the 
PLC polls the sensors and directs the actuators. 




The PLC processor, based on the control algo- 
rithms execution characteristics, determines the 
time behavior of the system. In a distributed sys- 
tem, the time behavior is determined by the time 
behaviors of the application execution character- 
istics on the local processors, the local protocol 
stacks, and the communication network. A dis- 
tributed system provides true multiprocessing 
and therefore the potential for improving com- 
putational throughput and synchronization char- 
acteristics of the system. 

In implementing distributed systems, the 
process control industry has developed a wide 
range of communication networks. The comput- 
er industry has also provided a range of networks 
for the general distributed computing environ- 
ment, with Ethernet being the most pervasive at 
the present time. Traditionally, Ethernet has not 
been used in control environments; it is found 
within the enterprise levels of process control and 
is increasingly being coupled to lower-level con- 
trol functions. Advertisements for field-level 
products using Ethernet are beginning to appear 
in trade publications. 

The use of general computer networks in 
field-level control will require some modification 
of techniques normally used in real-time systems. 
The computer industry has developed a number 
of distributed system techniques that will be use- 
ful in control. In particular, there is an increasing 
amount of literature on the use of real-time 
clocks in distributed systems. 

Clocks are found at the PLC level of current 
process control systems. However, time synchro- 
nization at the device level is generally limited to 
timers and time ricks distributed over the net- 
work from a PLC. Lets explore the use of true 
real-time clocks at all levels of distributed closed- 
loop control systems. 

Event or data driven? 

In an event-driven system, it is important 
that the occurrence of an event be made visible 
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For closed-loop control, it is imperative 
that control algorithms are provided with 
the correct sampling time information for 
each of the control variables. 



Terminology r ^ # >^ 



PLC 



PID 



logic controller 

proportional* 
integral-derivative 



CSMA carrier-sense 
multiple access 



CAN 



KTP 



controller area 



network time 
protocol 



in a reliable and timely manner. The order of 
events and their time relationship to the real 
world must be preserved. For events, time accu- 
racy and distribution latency are the prime tim- 
ing considerations. Alarms, state machines, and 
device commands are typical control structures 
using event mechanisms. 

In a data-driven system, such as continuous 
closed-loop control, the real-world time relation- 
ship of successive data points must be preserved. 
Distribution latency (delay) must be such that 
the response times and stability requirements of 
the loop are not left unmet. The system through- 
put must be adequate to process and distribute 
the data. 

In distributed control systems, each node must 
explicitly deal with synchronization issues. 
Messages without time stamps, when passed 
between nodes, can implement order, but not 
time specification. If time specification is to be 
imposed, then at least some of the nodes must 
have access to a clock For closed-loop control, it 
is imperative that control algorithms are provid- 
ed with the correct sampling time information 
for each of the control variables. 

Provide the time 

There are several ways to provide the sam- 
pling time information required by control 
algorithms like the widely used proportional- 
integral-derivative (PID) algorithm. 

Polling of the sensors by the node implement- 
ing the control algorithm is commonly used 
today. The polling is normally periodic as seen by 
the polling node. The time error in polled sys- 
tems is the departure from stricdy periodic time, 
termed time jitter y inherent in various compo- 
nents of the system. 



The main sources of this jitter are die r~~»- 
munication protocol stack and the operatin w 
terns of the nodes. In the case of the protocol 
stack, this jitter results from queuing proce ,ses 
and parsing protocol headers. 

Operating system time jitter arises from inter- 
nal operations such as context switching and ser- 
vicing software-maintained timers and clocks. 
With a carrier-sense multiple access (CSMA) 
protocol stack, such as Ethernet or LonTalk, this 
jitter can be important and is usually not known 
by the algorithm. However, even in non-CSMA 
stacks, such as controller area network (CAN) or 
Foundation fieldbus, care must be taken that 
operating system and application code jitter is 
negligible, as this jitter can be comparable to the 
jitter in a CSMA network. 

Since there is no way to measure this jitter for 
each packet, there is no way to correct for jitter in 
simple, polled systems. A common variation on 
this theme is for a central node to distribute 
"ticks," which other nodes use to synchronize 
sampling. These ticks are subject to the same jit- 
ter problems as polling. In polled systems, con- 
trol algorithm computations can only assume 
that the data was sampled at the poll times. 

Another alternative is to make the entir 
tributed system operate synchronously. Tht, ..as 
been done by running all the communications and 
computations on an enforced time-slice schedule. 
When run using a CSMA protocol stack, such a 
system can eliminate collision jitter, since by defi- 
nition there is no contention for the network. 

This type of setup uses synchronized clocks to 
enforce the schedule so data can be time 
stamped. Tune-slice systems based on token rings 
or their equivalent, rather than on synchronized 
clocks, will have uncorrected time jitter compara- 
ble to polled systems. 

The alternative presented here is to provide 
all nodes in the distributed system with accu- 
rately synchronized clocks. With such clocks, 
sensor data can be time stamped and control 
algorithms can use these time stamps to correct 
for the sample timing errors resulting from com- 
putational, operating system, and protocol stack 
jitter and delay. The accuracy of these clocks will 
determine the effectiveness of the corrections. To 
be effective, nodes must have hardware support 
for generating time stamps based on appropriate 
transducer control signals. Such systems can use 
CSMA protocols provided the clock synchro- 
nization is adequate. ( 

As a side benefit, time stamps provide ^a^ 
method for detecting and correcting for missing 
or out-of-order data. Both missing and out-of- 
order data are possible in networked distributed 
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200 picoseconds/bin 



Figure L lime differences between two docks using Ethernet communication 




L6 nanoseconds/bin 



Figure 1 lime differences between two docks using Lontalk communication 



systems, especially if routers are present. However, 
when CSMA networks are used under the same 
conditions (number of nodes and network traffic) 
as other fieldbus networks, the occurrence of 
missing and out-of-order data can be made rare 
and can be corrected in the control algorithm. 

Synchronize your clocks 

There are a number of techniques in use for 
synchronizing the clocks in a distributed system. 
A central time server is one possibility, but it intro- 
duces delays and excessive message traffic For sys- 
tems with more than a few nodes, a better choice 
is for each node to have a local dock parridpating 
in a synchronization protocol among the nodes. 

The computer industry uses protocols such as 
the network time protocol (NTP) for dock syn- 
chronization among PQ and workstations. NTP 
is based on message exchange and produces accu- 
racy on the order of a few milliseconds over 
ihernet networks. To attain better accuracy, some 
sort of hardware assistance is required. Systems 
exist that produce accuracy of 10 microseconds. 

We have experimented with dock synchro- 
nization based on hardware-assisted detection and 



recognition of special network packets for both 
Ethernet and LonTalk protocols. Synchronization 
accuracy is measured by recording the differences 
in the time stamps marking the occurrence of a 
series of mutually visible events as determined by 
the local docks in two nodes. 

Figure 1 is a histogram of these differences for 
two nodes communicating using the Ethernet 
protocol and demonstrates synchronization accu- 
racy on the order of 20 nanoseconds. Figure 2 is 
a histogram of these differences for two nodes 
communicating using the LonTalk protocol and 
demonstrates synchronization accuracy on the 
order of 100 nanoseconds. 

In both cases, this synchronization was' 
obtained with one packet per second devoted to 
the synchronization algorithm. Except during 
the start-up phase of the algorithm, the number 
of synchronization packets per second is inde- 
pendent of the number of docks being synchro- 
nized. The accuracy differences between the 
Ethernet and LonTalk implementations are due 
primarily to the differences in the network bit 
rates — 10 Mbps for Ethernet and 1.2 Mbps for 
LonTalk. 
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The key benefit of this clock synchronization 
method is increased accuracy. While millisecond 
accuracy obtainable by conventional methods is 
adequate for many closed-loop control and mon- 
itoring applications, the submicrosecond accuracy 
reported here allows fester systems to be con- 
trolled in a networked environment. 

Ethernet clocks the process 

A simple control system was built to test the 
feasibility of implementing synchronized clocks 
and CSMA networks. The system is illustrated in 
Figure 3. 




Figure 3. Test system block diagram 
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Ethernet 

Both the tachometer and controller electron- 
ics were implemented using Motorola 68331 
microprocessors. The communication mecha- 
nism is the unacknowledged datagram protocol 
on a lObase-T Ethernet. Each node contains a 
clock synchronized as described earlier. 

The controller is a standard PID with trial 
coefficients determined by the Ziegler-Nichols 
frequency response method The system was not 
stable using these initial values and was manually 
altered to produce a marginally stable one. Using 
marginal stability enabled testing under condi- 
tions that maximized the chances of observing 
deviations introduced by sampling jitter and net- 
work degradation. 

The system was operated both in a polled 
mode in which the controller requested samples 
and in a push mode in which the tachometer 
controlled the sampling time. The data was 
always time stamped based on the local clock in 
the tachometer. The controller could be config- 
ured to either use or ignore the time stamps when 
computing its output. 

As expected from previous analysis, it was diffi- 
cult, but not impossible, to induce noticeable 



degradation in the system. During the ' 
of the data, the packet load on the Etheuie 
25% of capacity as measured by a local-area n .t- 
work analyzer. In each case, 100 samples of t! .:e 
waveforms were collected. For each waveform, the 
root mean square deviation of the actual response 
from the mean value was computed as a oercent- 
age of the mean value. This calculation was done 
independendy for the falling and rising portions of 
the square wave perturbation. The results are seen 
in the table below. 

While more experiments are necessary to fully 
quantify these results, it appears that the settling 
rime was better using the time stamp corrections. 

Similar experiments were conducted in which 
packets were deleted to simulate network collisions 
or checksum errors with no visible degradation in 
performance. In all cases, the network packet load 
ranged from about 10% to 30% of capacity 

The only dramatic network-based interference 
produced was a result of bogus packets specifically 
addressed to a control system node. Such packets 
pass the usual Ethernet hardware filter and must be 
processed and rejected by the lower levels of the 
protocol stack, which steals processor cycles from 
the application. These experiments need to be 
repeated for controllers implementing notch r ^ 
and other algorithms that may show more 6. 
tivity to the timing-induced phase or amplitude 
jitter. 

Additionally, both the phase and amplitude 
noise introduced by rime jitter should be studied, 
particularly that introduced by alternative algo- 
rithms and conditions. Further, evidence exists 
that clock usage in event-driven applications needs 
to be examined as well. 

Its clear that the use of synchronized clocks in 
data-driven closed-loop control systems using 
Ethernet as the field bus is a viable solution. The 
ability to synchronize clocks to accuracy more than 
adequate for most control problems has been 
demonstrated on both the Ethernet and LonTalk 
protocols. The presence of these clocks allows 
algorithms to be corrected for the actual rimes of 
sampling, potentially eliminating computation, 
protocol stack, and communication time jitter. IT 





Percent deviation using 
time stamp values 


Percent deviation using ; 
nominal time values 


Falling half of square wave 


4.69 


4.73 


Rising half of square wave 


4.60 


4.62 
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